Button activation within an eye-controlled user interface

ABSTRACT

Systems and methods for activating a button within a display are described. Embodiments of the invention provide a multi-step activation process using user eye movement within the display. The multi-step activation process comprises displaying an confirmation element in response to a first user gaze at a button. The button is subsequently activated in response to a second user gaze at the confirmation element.

TECHNICAL FIELD

The present disclosure generally relates to systems and methods foreye-controlled user interfaces, and more particularly, to multi-stepactivation processes of buttons based on user gaze fixation within adisplay such as an augmented reality (“AR”) or virtual reality (“VR”)display.

BACKGROUND

The growth of AR/VR technologies across a large and diverse set ofmarkets is well understood by one of skill in the art. Markets such asgaming, media, search, and information management implement a variety ofdifferent AR/VR products to allow an individual to interact within avirtual environment. These AR/VR products provide an individual a richand dynamic platform in which the user can retrieve information, viewmedia content, navigate virtual scenes and interact with other types ofcontent in a manner unique to the AR/VR environment. It is importantthat these AR/VR products maintain a user-friendly experience throughouttheir use and provide a user the ability to interact with virtualcontent in a simple and consistently accurate manner.

AR/VR technologies are oftentimes constrained by the way an individualcan interact with the virtual environment. Some AR/VR products implementan eye-controlled user interface where a user interacts with virtualcontent using eye movement. This interface may rely exclusively oneye-controlled interactions or may use a combination of eye-controlledinteractions with other types of user control such as hand-gestures,hand controllers, head movement or other types of movement that istranslated into the virtual environment. Eye-controlled user interfaces,especially those that rely exclusively on eye movement, are limited to asmall subset of ways in which a user can interact with virtual contentand present unique challenges in providing a robust and reliableinterface. One such issue with eye-controlled user interfaces isensuring consistently accurate activation of buttons based on user eyemovement within the display.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a display illustrating potential false activation ofbuttons caused by user eye movement.

FIG. 2 illustrates an exemplary multi-step button activation processaccording to various embodiments of the present disclosure.

FIG. 3 show examples of a button and corresponding confirmation elementsaccording to various embodiments of the present disclosure.

FIG. 4 highlights gaze characteristics of location and time according tovarious embodiments of the present disclosure.

FIG. 5 illustrates various distance characteristics of confirmationelements to buttons according to various embodiments of the presentdisclosure.

FIG. 6 illustrates an exemplary multi-step button activation processwith drop-down confirmation element according to various embodiments ofthe present disclosure.

FIG. 7 illustrates an exemplary multi-step button activation processwith slider confirmation element according to various embodiments of thepresent disclosure.

FIG. 8 illustrates an exemplary multi-step button activation processwith spike-style confirmation element according to various embodimentsof the present disclosure.

FIG. 9 illustrates an exemplary multi-step button activation process inwhich a button border is initiated by a user gaze according toembodiments of the present disclosure.

FIG. 10 illustrates an enhanced button bar according to embodiments ofthe present disclosure.

FIG. 11A illustrates virtual response block according to embodiments ofthe present disclosure.

FIG. 11B illustrates virtual response blocks with and without enhancedbuttons according to various embodiments of the invention.

FIG. 12 illustrates exemplary button dismissals according to embodimentsof the present disclosure.

FIG. 13 illustrates further exemplary button dismissals according toembodiments of the present disclosure.

FIG. 14 illustrates a state space model showing a multi-step activationprocess for a button according to embodiments of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS

The figures and the following description relate to various embodimentsby way of illustration. It is noted that from the following discussion,alternative embodiments of the structures and methods disclosed hereinwill be readily recognized as viable by one of skill in the art. It isfurther noted that any headings used herein are only for organizationalpurposes and shall not be used to limit the scope of the description orthe claims. All documents cited are incorporated by reference herein intheir entirety.

Reference in the specification to “one or more embodiments,” “preferredembodiment,” “an embodiment,” “embodiments,” or the like means that aparticular feature, structure, characteristic, or function described inconnection with the embodiment is included in at least one embodiment ofthe disclosure and may be in more than one embodiment. Also, theappearances of the above-noted phrases in various places in thespecification are not necessarily all referring to the same embodimentor embodiments.

One skilled in the art shall recognize that: (1) certain steps mayoptionally be performed; (2) steps may not be limited to the specificorder set forth herein; (3) certain steps may be performed in differentorders; and (4) certain steps may be done concurrently.

For purposes of this disclosure, the term “button” is defined as asymbol within a display that can be activated by a user's gaze, examplesof which include (but are not limited to) icons, shapes, text, glyphs,links, menus and menu components, pointers, etc. The term “confirmationelement” is defined as a symbol within the display that appears inresponse to a user gaze fixating at a button. The confirmation elementis associated with the button and provides confirmation of a user'sintent to activate the button. The term “gaze” is defined as a userlooking at a location within the display. The term “fixate” is definedas a period of time of a user gaze. The period of time during which auser fixates at a location may be measured in relation to a threshold.This threshold may be static or vary based on factors such as userhistory, user activity, machine learning processes, saccade predictionor other factors that adjust this threshold. The time may be measured asan incremental counter or a countdown time.

Embodiments of the present invention implement an eye-controlled userinterface in which buttons are activated using a multi-step activationprocess. Eye-controlled user interfaces track a user's eye movement andmap the movement within a virtual environment. In one example, AR/VRgoggles have an inward-facing camera that monitors user eye movement ashe/she is viewing a display. The goggles correlate this eye movement tothe virtual content being viewed to allow interaction between the userand the display. In another example, contact lenses are worn by a userand track the movement of the user's eyes. This eye movement iscorrelated to a display projected onto the user's eye from aprojector(s) embedded in the contact lens, examples of which may befound in U.S. patent application Ser. No. 16/940,152, filed on Jul. 27,2020, entitled “Eye-Tracking User Interface for Virtual Tool Control,”listing inventors Haine et al.; U.S. patent application Ser. No.16/005,379, filed on Jun. 11, 2018, entitled “Contact lens gaze trackingarchitectures,” listing inventors Mirjalili et al.; and U.S. patentapplication Ser. No. 16/200,039, filed on Nov. 26, 2018, entitled“Eye-mounted Displays Including Embedded Solenoids,” listing inventorsMirjalili et al., which patent documents are incorporated by referenceherein in their entirety and for all purposes.

Eye-controlled user interfaces (either exclusively or partiallycontrolled by eye movement) present unique challenges in defining howbuttons are activated within a display. One such challenge is definingthe user's intent within the display when he/she looks at a button. Thesystem should be able to differentiate between when a user looked at abutton to activate it and when the user viewed the button withoutintending to activate it.

FIG. 1 illustrates an aspect of unintended activation of buttons withina display. As shown, an exemplary display is provided and defined as afield of view 110 within an AR/VR environment. The field of view 110includes a variety of different eye-controlled buttons 115-121 thatallow the user to interact with the display. The user's eye movement istracked within a two-dimensional space 125 and mapped in the field ofview to allow the system to respond to eye movement and manage virtualcontent.

Accurate activation of buttons within the field of view 110 ischallenging if the system relies on a user looking at a button toactivate it. Correlating a user's intent to activate a button versus auser perusing content in the display is essential to a user interfacethat relies on eye movement as a primary means for user input. As a userviews the display, his/her eyes may look at one or more of the buttonswithin the display for a short period of time without intending toactivate a button. For example, a user's eyes may simply traverse 130across the field of view 110 to view content displayed within virtualenvironment. This eye movement is mapped into the display and may crossany number of buttons 117, 118. If these buttons are activated, thedisplay may become unmanageable as buttons are unintentionallyactivated. Various embodiments of the invention address this problem bydefining button activation processes that more accurately predict theintent of a user to activate a particular button.

FIG. 2 illustrates exemplary eye-controlled, multi-step buttonauthentication processes according to various embodiments of theinvention. A three-step activation process is illustrated that allows auser to activate a button 215 with eye movement. A user fixates a firstgaze 210 at a location associated with the button 215. This location maybe within the borders of the button 215, at an edge of the button 215,or within a predefined distance external to the edge of the button 215.This location may account for variations within the user's gaze such ajitter or drift as the user is looking at the location. In response tothe first gaze fixation 210, a confirmation element 220 is generatedwithin the display, three examples of which are illustrated herein.

The confirmation element 220 is generated within the display in a mannerto visually show an association with the button 215 to the user. In afirst example, the confirmation element 220 is a triangle placed at anedge of the button. In a second example, the confirmation element 220 isa spike coupled to the button. In a third example, the confirmationelement 220 is a virtual slider of the button. The shape, location,shade and/or color of the confirmation element may vary based on designimplementations. The three confirmation elements are described in moredetail later in this description.

The user fixates a second gaze 250 at the confirmation element toactivate the button. This second gaze 250 may be at a location withinthe confirmation element, at an edge or point of the confirmationelement 250 or within a predefined distance external to the confirmationelement 250. The length of time of the second gaze fixation 250 may beequal to or different from the first gaze 210. After performing thesecond gaze fixation 250 at the confirmation element 220, thecorresponding button is activated. This multi-step activation process ofa button reduces unintended activation of buttons based on user eyemovement over prior art systems.

FIG. 3 illustrates representative visual relationships between a buttonand confirmation element according to various embodiments of theinvention. As previously stated, the confirmation element 310 isdisplayed so that a user viewing the control will associate it with thebutton 305. There are a variety of confirmation element displaycharacteristics that show this association to the button 305.

A first characteristic is a confirmation element location 320 relativeto the button 305. In one example, the confirmation element 350 ispositioned at a border of the button 305. If the confirmation element350 appears after a user gaze fixates at the button 305, the appearanceof the confirmation element 350 within the visual area at which the useris gazing will indicate an association with the button 305. Accordingly,one skilled in the art will recognize that a confirmation element 350may be positioned within a button, at an edge or border of a button, orwithin a distance proximate to the button such that a user will see theconfirmation element when it is initiated.

A second characteristic is a user-guided confirmation element 330. Inanother example, the confirmation element 360 is visually coupled to thebutton 305 such that the user will recognize the association betweenbutton and confirmation element. This visual coupling may be implementeda number of different ways within a display such as a line 325connecting the button 305 to the confirmation element 360. The coupledconfirmation element 360 may be positioned at a variety of locationsexternal to the button 305.

Other characteristics such as related visual shapes, border highlights,shading, etc. 340 may also be used to visually associated a confirmationelement with a button. In yet another example, a slider confirmationelement 370 is visually displayed proximate to the button 305. Thisslider 370 may be shaded or have a different color. It may also have asimilar shape to the button or appear to extend the button onceactivated. Other confirmation element characteristics may also be usedto visually associate a button with a confirmation element.

FIG. 4 illustrates location and fixation time characteristics of a usergaze according to various embodiments of the invention. A user gazeoccurs when a user fixates at a location within a display for a timeequal to or exceeding a threshold. For purposes of triggering thedisplay of a confirmation element or activating a button, the user gazemay appear within a button/confirmation element 410, at an edge or lineof a button/confirmation element 420 or outside of a button/confirmationelement at a set distance. A threshold of time 430 is applied to thegaze fixation at the location. Once the time threshold 430 is reached,the system generates a response to the gaze fixation. This timethreshold 430 may be a static period of time or may change based on avariety of factors such as the user's prior activity, machine learningprocesses that predict a user's likelihood of intending an activationprocess, the frequency a button has been activated, etc. This time maybe measured by a clock or counter that increments forward or a clockthat countdowns from a first value to zero. The clock may be initiatedusing a variety of factors such as when a user gaze begins looking at aparticular location, when a user starts to saccade to a particularlocation, etc.

FIG. 5 provides examples of location distances between a button and aconfirmation element according to various embodiments of the invention.In each example, a confirmation element 510, 515, 520 is positionedproximate to its corresponding button 530, 535, 540. The location of aconfirmation element may be defined at a distance 550, 555, 560 thatallows a user to associate a confirmation element with a correspondingbutton. For purposes of this disclosure, the distance should be smallenough so that the confirmation element appears within the user's fieldof view after a user gaze fixates at the button for a certain period oftime. Specifically, the user should not be required to scroll beyond thefield of view to subsequently gaze at the confirmation element. Aspreviously mentioned, the confirmation element may be displayed suchthat an association to the button is apparent to the user such asplacing the confirmation element 510 on an edge of the button, placingthe confirmation element 515 within an extension of the button orplacing the confirmation element 520 at the end of a line coupled to thebutton.

FIG. 6 illustrates a first implementation of a multi-step buttonactivation using eye movement according to various embodiments of theinvention. At a first period of time T₁ 610, a button 620 is visiblewithin a display and a user is looking away from the button 620. At asecond period of time T₂ 625, a user gaze fixates at the buttonresulting in a confirmation element 630 appearing proximate to a loweredge of the button 620. At a third period of time T₃ 635, the user gazefixates at the confirmation element 630. In certain embodiments, theconfirmation element size increases 637 at the beginning of the usergaze. At a fourth period of time T₄ 640, the button 620 is activated andthe user can interact with the activated button or any contentassociated with the activated button. In some examples, the confirmationelement 630 disappears after the button 620 is activated. In otherexamples, the confirmation element 630 remains visible after the button620 is activated and may provide further functionality such asdeactivating the button 620 or deleting content displayed within thevirtual environment.

FIG. 7 illustrates a second implementation using a slider-style,multi-step button activation using eye movement according to variousembodiments of the invention. At a first period of time T₁ 710, a button720 is visible within a display and a user is looking away from thebutton 720. At a second period of time T₂ 725, the user gaze fixates atthe button resulting in a confirmation element 730 appearing as a shadedslider adjacent to the button 720. At a third period of time T₃ 735, theuser gaze fixates at the slider confirmation element 730. At a fourthperiod of time T₄ 740, the button 720 is activated and the user caninteract with the activated button, or any content associated with theactivated button. In certain examples, the slider confirmation element730 changes color or shade 745 to indicate activation of the button 720or the button 720 changes color or shade 750 to indicate activation.

FIG. 8 illustrates a third implementation using a spike-style,multi-step button activation using eye movement according to variousembodiments of the invention. At a first period of time T₁ 810, a button820 is visible within a display and a user is looking away from thebutton 820. At a second period of time T₂ 825, the user gaze fixates atthe button resulting in a confirmation element 830 appearing as anend-point spike coupled to the button 820 via line 833. At a thirdperiod of time T₃ 835, the user gaze fixates at the spike confirmationelement 830. At a fourth period of time T₄ 840, the button 820 isactivated and the user can interact with the activated button, or anycontent associated with the activated button. In certain examples, thespike confirmation element 830 changes color or shade 845 to indicateactivation of the button 820 or the button 820 changes color or shade850 to indicate activation.

FIG. 9 illustrates a fourth implementation using an enhanced border,multi-step button activation using eye movement according to variousembodiments of the invention. At a first period of time T₁ 910, at leasta portion of button 920 is visible within a display and a user islooking away from the button 920. Portions of the button 920, such asthe border of the button, may not be visible at this time for a varietyof purposes including to reduce the button size shown within thedisplay. At a second period of time T₂ 925, the user gaze fixates at thebutton resulting in a complete display of the button 927 and aconfirmation element 930 appearing at an edge of the button 920. At athird period of time T₃ 935, the user gaze fixates at the confirmationelement 930. In certain embodiments, the confirmation element may changeshape or color, such as increase size 937, when this gaze begins. At afourth period of time T₄ 940, the button 920 is activated and the usercan interact with the activated button, or any content associated withthe activated button.

This reduced button footprint allows the display to show more buttons inan organized and uncluttered manner. It also allows buttons to bepositioned closer to each other within the display. Exemplaryimplementations of this enhanced border button are provided below.

FIG. 10 illustrates a menu within a display having enhanced bordermulti-step activation according to various embodiments of the invention.A menu 1010 is shown having four icons 1015-1019. In this example,borders of the four buttons are not visible when a user is not looking1020 at it. When the user fixates a first gaze at a first button 1015, aborder 1031 appears around the button 1030 and a confirmation element1032 is displayed. As previously mentioned, the button 1030 appearancemay change in a variety of ways in response to a user gaze fixation; allof which fall within the scope of the invention.

This concept of enhanced button is shown across the other buttons in themenu. A second gaze fixation at a second button 1016 results in a border1041 and confirmation element 1042 being displayed. The same processoccurs at a third button 1017 and a fourth button 1018. Using theseenhanced buttons allows the menu 1010 to have button located closertogether and displayed in an organized manner. The buttons 1015-1019 maybe activated by performing a subsequent gaze fixation at theconfirmation element.

FIG. 11A shows an exemplary virtual response block comprising twobuttons with enhanced confirmation elements according to variousembodiments of the invention. The virtual response block 1110 allows auser to answer a question by selecting a BEGIN button 1120 or a CANCELbutton 1130. The two buttons are located in close proximity to eachother within the virtual response block 1110 and neither has a borderuntil a user gaze occurs.

A user gaze fixates at the CANCEL button 1130 resulting in a border 1135and confirmation element 1136 being displayed. In this instance, theenhanced button within the border extends beyond the virtual responseblock 1110 and is closer to/overlaps with the BEGIN button 1120. Amulti-step activation of the CANCEL button 1130 occurs when the userfixates a subsequent gaze at the confirmation element 1136. A similarmulti-step activation process occurs with the BEGIN button 1120.

For illustrative purposes, a version of the virtual response block 1110is provided in which both a BEGIN button 1160 and a CANCEL button 1170are shown with borders. If borders are shown for both buttons, overlapbetween buttons and borders with the virtual response block 1110. Oneskilled in the art will recognize a resulting more-efficient use ofspace within a display when enhanced buttons are used. A variety ofdifferent implementations of enhanced buttons may be used where a firstportion of the button is only visible and a second portion of the button(e.g., border) appears when a user gaze fixation occurs.

FIG. 11B further illustrates the improved virtual footprint of enhancedbuttons versus buttons within a response block according to variousembodiments of the invention. A first version with button having bordersis shown in a first block 1180. A second version with enhanced button isshown in a second block 1190. One skilled in the art will recognize thebenefits of enhanced buttons including more efficient use of space, theimproved ability to organize a display, etc.

FIG. 12 illustrates button dismissal using multi-step authenticationaccording to various embodiments of the invention. A virtual dismissalblock 1210 is shown with a CLOSE button 1215. In response to a userfirst gaze fixating proximate to the CLOSE button 1215, a confirmationelement 1220 appears. A second gaze fixates proximate to theconfirmation element 1220 results in the dismissal block 1210disappearing. In addition, content associated with the dismissal block1210 may also be removed and/or transformed in response to theactivation of the dismissal.

As previously described, the confirmation element may be provided in avariety of locations and shapes. A second example of a confirmationelement is shown where a glyph, such as an X, 1240 is provided at acorner of a dismissal block 1230. Embodiments of the invention mayimplement different enhancements to the confirmation element wherein itappears/disappears, changes size, changes shape or color, etc. toconfirm a user's intent to activate or dismiss a button.

FIG. 13 illustrates enhanced button dismissal using multi-stepauthentication according to various embodiments of the invention. In afirst example, a block 1310 is provided with a glyph 1311 at the uppercorner that indicates the block may be dismissed. A user may start themulti-step dismissal by fixating at the glyph 1311. In response, aborder 1312 and a confirmation element 1315 is displayed. A usersubsequently gazes at the confirmation element 1315 to complete themulti-step dismissal resulting the in the block 1310 being removed fromthe display. Content related to the dismissal block may also be removedfrom or transformed within the display.

The confirmation element may be further enhanced by having a glyphchange its size, shape, color or shade as part of the activationprocess. As shown, a block 1350 may have a dismissal glyph 1355 at acorner. As a user gaze fixates at the dismissal glyph 1355, it maydisplay a border 1358 and confirmation element 1360, and increase itssize 1370 to further enhance the next step in the dismissal process.

FIG. 14 illustrates an exemplary state space model of a multi-stepactivation process of a button according to various embodiments of theinvention. In an initial state 1410, a button is not enabled norconfirmed as the user is looking away from the button or fixating on thebutton for a period less than a time threshold T₁. The buttontransitions to a second state 1420 after a user fixates a gaze at thebutton for a period greater than the time threshold T₁. In this secondstate 1420, the button is enabled but not confirmed and a confirmationelement appears.

The button transitions to a third state 1430 after a user fixates a gazeat the confirmation element for a period greater than the time thresholdT₂. In this third state 1430, the button is enabled and confirmedresulting in an activation of the button, activation of contentassociated with the button and/or other response/command 1440 executedby activating the button. The button may also visually change toacknowledge confirmation.

If the button in the second state 1420 fails to transition to the thirdstate 1430 within a predefined period of time, the button transitions tothe initial state 1410.

It will be appreciated to those skilled in the art that the precedingexamples and embodiments are exemplary and not limiting to the scope ofthe present disclosure. It is intended that all permutations,enhancements, equivalents, combinations, and improvements thereto thatare apparent to those skilled in the art upon a reading of thespecification and a study of the drawings are included within the truespirit and scope of the present disclosure. It shall also be noted thatelements of any claims may be arranged differently including havingmultiple dependencies, configurations, and combinations.

What is claimed is:
 1. A method for activating an enhanced button, themethod comprising: displaying a first portion of the enhanced buttonwithin a display; identifying a first user gaze at a first locationwithin the display, the first location being associated with theenhanced button; monitoring a first fixation time of the first user gazerelative to a first threshold; in response to the first fixation timebeing longer than the first threshold, displaying an a second portion ofthe enhanced button and an confirmation element associated with theenhanced button within the display; identifying a second user gaze at asecond location within the display, the second location being associatedwith the confirmation element; monitoring a second fixation time of thesecond user gaze relative to a second threshold; and in response to thesecond fixation time being longer than the second threshold, activatingthe enhanced button.
 2. The method of claim 1 wherein the second portionof the enhanced button is a border.
 3. The method of claim 1 wherein theenhanced button is a dismissal button.
 4. The method of claim 3 whereinthe confirmation element is positioned proximate to a corner of theenhanced button.
 5. The method of claim 1 wherein the confirmationelement changes at least one of a shape, size and color in response toan initiation of the second gaze.
 6. The method of claim 1 wherein theenhanced button is located in a menu within the display.
 7. The methodof claim 6 wherein the second portion of the enhanced button overlaps anadjacent button within the menu.
 8. The method of claim 1 wherein thesecond portion of the enhanced button overlaps a border of a virtualblock containing the first portion of the enhanced button.
 9. The methodof claim 1 wherein user eye movement is tracked using a contact lenshaving a plurality of sensors.